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Abstract. We consider a new kind of straight and shifted plane partitions/ Young tableaux — ones whose 
diagrams are no longer of partition shape, but rather Young diagrams with boxes erased from their upper 
right ends. We find formulas for the number of standard tableaux in certain cases, namely a shifted staircase 
without the box in its upper right corner, i.e. truncated by a box, a rectangle truncated by a staircase and 
a rectangle truncated by a square minus a box. The proofs involve finding the generating function of the 
corresponding plane partitions using interpretations and formulas for sums of restricted Schur functions and 
their specializations. The number of standard tableaux is then found as a certain limit of this function. 
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1. Introduction 

A central topic in the study of tableaux and plane partition are their enumerative properties. The number 
of standard Young tableaux (SYT) is given by the famous hook-length formula of Frame, Robinson and 
Thrall, a similar formula exists for shifted standard tableaux. However, most tableaux and posets in general 
do not have such nice enumerative properties, for example their main generalizations as skew tableaux are 
not counted by any product type formulas. 

In this paper we find product formulas for special cases of a new type of tableaux and plane partitions, 
ones whose diagrams are not straight or shifted Young diagrams of integer partitions. The diagrams in 
question are obtained by removing boxes from the north-east corners of a straight or shifted Young diagram 
and we say that the shape has been truncated by the shape of the boxes removed. We discover formulas for 
the number of tableaux of specific truncated shapes: shifted staircase truncated by one box in Theorem [l] 
rectangle truncated by a staircase shape Theorem [2] and rectangle truncated by a square minus a box in 
Theorem [3j these are illustrated as 



The proofs rely on several steps of interpretations. Plane partitions of truncated shapes are interpreted as 
(tuples of) SSYTs, which translates the problem into specializations of sums of restricted Schur functions. 
The number of standard tableaux is found as a polytope volume and then a certain limit of these specializa- 
tions (generating function for the corresponding plane partitions). The computations involve among others 
complex integration and the Robinson-Schensted-Knuth correspondence. 

The consideration of these objects started after R. Adin and Y. Roichman asked for a formula for the 
number of linear extensions of the poset of triangle-free triangulations, which are equivalent to standard 
tableaux of shifted straircase shape with upper right corner box removed, 
formula in question, namely 

Theorem 1. The number of shifted standard tableaux of shape 6 n \ Si is equal to 



AR . We find and prove the 
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We also prove formulas for the cases of rectangles truncated by a staircase. 

Theorem 2. The number of standard tableaux of truncated straight shape (n, n, 
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Theorem 3. The number of standard truncated tableaux of shape n r ' 

/ nm — k 2 + 1 N 
m(n — k) 

We will also exhibit connections with boxed plane partitions, as the generating function we use is the 
same as the volume generating function for boxed plane partitions. 

We should note that numerical data suggests there are almost no other truncated shapes with product- 
type formulas. We see this because the number of standard tableaux of a shape of size n has very large prime 
factors (e.g. on the order of 10 6 ) even for small (below 50) values of n. For example, there does not seem 
to be a product type formula for the immediate generalizations of the shapes above, e.g. shifted truncated 
shape S n \Sk for k > 1, or rectangles truncated by squares minus a staircase n m \ (k r , k — 1, . . . ,r + l,r). 
The lack of hypothetical general formulas prevents us from applying the usual, generally induction based, 
techniques used for proving the hook-length formulas enumerating standard Young tableaux. 

It might be interesting to investigate what other properties of SYTs, besides product-form formulas, have 
analogues in tableaux of truncated shapes. The presence of such formulas might suggest connections to 
representation theory and/or combinatorial properties like RSK. While almost all other truncated shapes 
lack product-type formulas, it is conceivable that other closed form formulas, e.g. determinantal, could exist. 



In AKR11 Adin, King and Roichman have independently found a formula for the case of shifted staircase 



truncated by one box and rectangle truncated by a square minus a box by methods different from the methods 
developed here. 

2. Background and definitions 

We will review some basic facts and definitions from the classical theory of symmetric functions and Young 
tableaux which will be used later in this paper. The full description of these facts, as well as proofs and 
much more can be found in Sta99 and Mac95 . 

An integer partition A of n is a weakly decreasing sequence of nonnegative integers A = (Ai, A2, • ■ • , A;) 
which sum up to n, i.e. Ai + • • • + A; = n. The length of A, ^(A), denotes the number of parts of A, in this 
case it is I. Partitions are depicted graphically as Young diagrams, a left justified array of n = |A|boxes, 
such that row i counted from the top has Ai boxes. For example, if A = (5, 3, 2) then l(X) — 3 and it's 



Young diagram is 



We will mean the Young diagram of A when we refer to A as a shape. A skew 



partition (and thus shape) A//x is the collection of boxes which belong to A but not fi when drawn with 



coinciding top left corners. E.g. the diagram of (5, 4, 3, l)/(3, 2) is 



A semi-standard Young tableaux (SSYT) of shape A and type a — («i, «2, ■ ■ •); where a%+a2+- ■ ■ = 
|A| is a filling of the boxes of the Young diagram of A with integers, such that the numbers weakly increase 
along each row and strictly decrease down each column, and there are exactly numbers equal to i. A 
standard Young tableaux (SYT) is an SSYT of type (1™) = (!,...,!). A plane partition of shape A is 
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a filling with nonnegative integers of the diagram of A, such that the numbers decrease weakly along rows 
and down columns. The same definitions carry over verbatim when the shape is skew, i.e. A/^t. 
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SSYT of type (2,2,3,3,2,1) 
The Schur function s\/^(xi, . 



SYT of shape (6, 4, 3) 
. , xi ) is defined as 



plane partition 



s\/^,{xx,...,xi) = ^2 xT ■> 

T:sh(T)=X/fi 

where the sum is over all SSYTs T of shape A//x and entries 1,2, ... ,1 and x T = Y[ %i 
Another definition for Schur functions of shape A is Schur's determinantal formula: 

s\(xi, . . .,x k ) 



if the type of T is a. 



a\+6,(xi, . . . ,xi) 



det[a- ; - j i f 



where 



){x\, . . . ,xi) = det[x^ 3 ] \ - =1 and (5; = (Z,Z — 1, . . . , 1). The Schur functions belong to the ring of 
symmetric functions, i.e. functions invariant under permutations of the variables. Other symmetric functions 
we will refer to in this paper are the homogenous symmetric functions h\{x\, . . . , xi) defined as 

hp ^ ^ Xi 1 Xi 2 • • • Xi l , h\ — J^J h\i • 

l<i\<i2---<ii<l 2=1 

These functions satisfy a lot of remarkable identities, some of which we will mention and use later here. 
We have that 

k 

(2) Y^hi{x 1 ,...,x k )t l 
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The Schur functions satisfy Cauchy-type identities, namely 
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y2s x (x 1 ,...)s x (y 1 ,...) = T[- 1 and V s x (x u . . .) = TT TT — 1 



While these identities can proven algebraically, the more elegant proof is via a bijection between pairs of 
SSYTs of same shape (P,Q), sh(P) = sh(Q) and matrices A = [a^] of nonnegative integer entries — the 
Robinson-Schensted-Knuth correspondence. We refer the reader to Sta99 for a full description of the 
bijection, proofs and properties. The most basic property is that if the row sums of A are a, i.e. a%j — on, 
and the column sums are /3, i.e. ^ = /3j, then the type of P is /? and the type of Q is a. RSK translates 
many combinatorial properties. For example, if A is a permutation matrix corresponding to the permutation 
w, and sh(P) = sh(Q) — X, then the length of the longest increasing subsequence of w is equal to Ai and 
the length of the longest decreasing subsequence is equal to l(X). This fact extends to general matrices 

/ ^ 

(Schensted's theorem), where we replace the permutation by a double array listing 



each aij times in 



of 



order of increasing is and then js and 'increasing subsequence' is replaced by 'weakly increasing subsequence 
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We will illustrated the statements above by an example. Let A = 



s' and 'decreasing subsequence' is replaced by 'strictly decreasing' in both i and j. 

then the corresponding 



double array is wa = 



Applying RSK to A we get (P, Q) with 
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= sh(P) = sh(Q) = (4, 2, 1) and a longest increasing subsequence of wa is indeed of 
1 1 1 2 2 3 3\ 

3 3 2 2 1 2/ anC ^ t ^ ie ^ on S est decreasing subsequence is of length = 3: 



We have that A 

length 4 = Ai: , , 

1 1 1 2 2 3 3' 
1 3 3 2 2 1 2, 

An important and remarkable combinatorial property of standard Young tableaux is the existence of a 
product type formula for the number of SYTs of a given shape A, denoted f\. The so called hook-length 
formulas for the number of straight standard Young tableaux (SYT) of shape A are: 
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and the number of standard Young tableaux of shifted shape A 



9\ = 
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We are now going to define our main objects of study. 

Let A = (Ai, A 2 , . . .) and [i = (^i,yu 2 , . . .) be integer partitions, such that Ai > A straight diagram 
of truncated shape A \ [i is a left justified array of boxes, such that row i has Ai — ^ boxes. If A has no 
equal parts we can define a shifted diagram of truncated shape A \ /i as an array of boxes, where row i 
starts one box to the right of the previous row i — 1 and has Ai — fii number of boxes. For example 



Do 



Di is of straight truncated shape (6, 6, 6, 6, 5) \ (3, 2) and D 2 is of shifted truncated shape (8, 7, 6, 2) \ (5, 2). 

We define standard and semistandard Young tableaux and plane partitions of truncated shape the usual 
way except this time they arc fillings of truncated diagrams. A standard truncated Young tableaux of 
shape A\y(i is a filling of the corresponding truncated diagram with the integers from 1 to |A| — \fj,\, such that 
the entries across rows and down columns are increasing and each number appears exactly once. A plane 
partition of truncated shape A \ \i is a filling of the corresponding truncated diagram with integers such 
that they weakly increase along rows and down columns. For example 
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are respectively a standard Young tableaux (SYT), a semi-standard Young tableaux (SSYT) and a plane 
partition (PP) of shifted truncated shape (5,4,2) \ (2). We also define reverse PP, SSYT, SYT by revers- 
ing all inequalities in the respective definitions (replacing weakly /strictly increasing with weakly/strictly 
decreasing). For this paper it will be more convenient to think in terms of the reverse versions of these 
objects. 

In this paper we will consider truncation by staircase shape 5k — {k, k— 1, k — 2, . . . , 1) of shifted staircases 
and straight rectangles. We will denote by T[i, j] the entry in the box with coordinate (i,j) in the diagram 
of T, where i denotes the row number counting from the top and j denotes the j— th box in this row counting 
from the beginning of the row. 

For any shape D, let 

F D (q)= £ g£T[iJ] 

T : PP, sh(T) = D 



be the generating function for the sum of the entries of all plane partitions of shape D. 
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3. A BIJECTION WITH SKEW SSYT 



We will consider a map between truncated plane partitions and skew Semi-Standard Young Tableaux 
which will enable us to enumerate them using Schur functions. 

As a basic setup for this map we first consider truncated shifted plane partitions of staircase shape S n \S k . 
Let T be such a plane partition. Let A- 7 = (T[l, j],T[2, j], . . . ,T[n — - the sequence of numbers in the 



jth diagonal of T. For example, if T = 
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then A 1 



I, 6, 4, 1), A 2 = (6, 4, 2) and A 3 = (5, 3). 



Let P be a reverse skew tableaux of shape X 1 /\ n ~ k , such that the entries filling the subshape A- 7 '/A- 7 ' +1 
are equal to j, i.e. it corresponds to the sequence A™ C A n_fe_1 C • • • C A 1 . The fact that this is all 
well defined follows from the inequalities that the T[i,j\s satisfy by virtue of T being a plane partition. 
Namely, A J+1 C A J , because \{ — T[i,j] > T[i,j + 1] = Clearly the rows of P are weakly decreasing. 

The columns are strictly decreasing, because for each j the entries j in the ith row of P are in positions 
T[i,j + 1] + 1 to T[i, j] < T[i — 1, j + 1] + 1, so they appear strictly to the left of the js in the row above 
('-!)• 

Define <fr(T) = P, <j> is the map in question. Given a reverse skew tableaux P of shape A \ fj, and 
entries smaller than n we can obtain the inverse shifted truncated plane partition T = <f>^ 1 (P) as T[i,j] = 
max(s|P[i, s] > j); if no such entry of P exists let s = 0. 

For example we have that 
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Notice that 



(5) 



]TTM=£PM + |A"- fe |(n-fc), 



where sh(P) = A7A"- fe . 

The map <p can be extended to any truncated shape, then the image will be tuples of SSYTs with certain 
restrictions. For the purposes of this paper we will extend it to truncated plane partitions of shape (n m ) \ 8k 
as follows. 

Let T be a plane partition of shape n m \Sk and assume that n < m (otherwise we can reflect about the main 
diagonal). Let A = (T[l, 1], T[2, 2], . . .,T[n,n]), fj, = {T[l,n- k],T[2,n- k + 1], . . .,T[k + l,n]) and let Tj be 
the portion of T above and including the main diagonal, hence of shifted truncated shape 5 n \5 k , and T 2 the 
transpose of the lower portion including the main diagonal, a shifted PP of shape (m, m — 1, . . . , m — n + 1). 

Extend 4> to T as <j)(T) = (0(7i), </>(T 2 )). Here <j>{T 2 ) is a SSYT of at most n rows (shape A) and filled 
with [1, . . . , to] the same way as in the truncated case. 

As an example with n = 5, m = 6, k = 2 we have 
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We thus have the following 
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Proposition 1. The map <j> is a bijection between shifted truncated plane partitions T of shape 5 n \ 5k filled 
with nonnegative integers and (reverse) skew semi-standard Young tableaux with entries in [1, . . . , n— k — 1] 
of shape X/^i with l(X) < n and l(fi) < k + 1. Moreover, ■ T[i,j] = J2i j P[hj] + H( n — Similarly 4> is 
also a bijection between truncated plane partitions T of shape n m \ 5k and pairs of SSYTs (P, Q) , such that 
sh(P) = X/fj,, sh(Q) = A with /(A) < n, l(fi) < k + 1 and P is filled with [1, . . . , n - k - 1], Q with [1, ...,m\. 
Moreover, J2T[i,j]=J2P[i,j]+J2Q[iJ]-\\\ + \tx\(n-k). 



4. SCHUR FUNCTION IDENTITIES 

We will now consider the relevant symmetric function interpretation arising from the map </>. Substitute 
the entries 1, . . . in the skew SSYTs in the image with respective variables x\, . . . and z\, . . .. The idea is to 
evaluate the resulting expressions at certain finite specializations for x and z{ e.g. x — (q,q 2 , ■ ■ ■ ,q n ~ k ~ 1 ) 
and z = (l,q,q 2 , . . . ,<7™ 1-1 )) to obtain generating functions for the sum of entries in the truncated plane 
partitions which will later allow us to derive enumerative results. 

For the case of shifted truncated shape <5„ \ 5k we have the corresponding sum 

(6) S n ,k(x;t) = ^ s \/tJ.( x i, ■ ■ ■ ,x n ^k~i)t^K 

A,^i|Z(A)<n,Z(/i)<fc+l 

and for the straight truncated shape n m \ 5k 

(7) D n ^k(x;z-t) = 53 s\{z)s\/»(x)t M - 

\,n\l(\)<n,l(jj,)<k+l 

We need to find formulas when Xi = for i > n — k — 1 and Z{ = for i > m. Keeping the restriction 
< HI we have that sx/^x) — if l(X) > n and this allows us to drop the length restriction on A in 
both sums. 

From now on the different sums will be treated separately. Consider another set of variables y = 
(yi, . . . , yk+i) which together with (xi, . . . , x n -k-i) form a set of n variables. Using Cauchy's first iden- 
tity in ([3]) we have that 

53 s\(x!, . . .,x n -k-i,tyi, . . -,ty k+ i) 

\\l(\)<n 

~n n 1 - XlX n 1 _ t2 n 1 _ Xity n x _ tyi > 

i<3<n—k— 1 J t<]<k+l J i, j J 

where the length restriction drops because s\(u\, . . . ,u n ) — when ^(A) > n. On the other hand we have 
that for any A s\(xi, . . . , x n - k -i, tyi, . . . ,yh+\t) = E M |i( Al )<fc+i s a/ m (>i, ■ ■ • , Xn-k-^s^tyi, . . .,ty k+ i), since 
again s ll (yt) = if > k + 1. We thus get 

(8) n 1 _ Xi n 1 _ — n 1 _ t2 n 1 _ x ty n 1 _ ty 

i<j<n—k—l J i<]<k+l J 1,3 J 

= $3 s \/n( x i,---,Xn-k-i)s^(yi,...,yk+i)t 1 ^ 

\,fi\l(fi)<k+l 

We now need to extract the coefficients of s^y) from both sides of Q to obtain a formula for S ni k[x; t). To 
do so we will use the determinantal formula for the Schur functions, namely that 

a v+5p {u) det[u^-%^ 

s v {ui,...,u p ) — 



a Sp (u) detK" J ]f 
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We also have that as p (zi, . . . , z p ) = YliKji 2 ^ - Zj). Substituting s p (y) with a Sk+1+Pl {y)/as k+1 (y) in the 
right-hand side of ([8]) and multiplying both sides by ag k+1 (y) we obtain 



(9) ^s A/A1 (x 1 , . . .,x n _ k _ 1 )a li+&k+1 {y 1 , . . .,y k +i)t M 

= TT 1 TT 1 TT ViZM TT 1 TT 1 



Observe that a Q (iti, . . . , u p ) = X^gs s S n ( u; )' u 5« 1 i ' ' ' u w P P w ith a,; > a^+i has exactly p! different monomials 
in y, each with a different order of the degrees of iij (determined by Moreover, if a ^ /3 are partitions of 
distinct parts, then a a (u) and a@(u) have no monomial in common. Let 

A ^ = £ Z "s^Xi • • • «S5 ■ 

For every /3 of strictly decreasing parts, every monomial in ap(u) appears exactly once and with the same 
sign in A(u), so a ( 3(w)A(u~ 1 ) has coefficient at vP equal to p\. Therefore we have 

(10) [y°] 8 */m( Xi ' ■ • • ' x n-k-i)a fl+Sk+1 [yi,..., y k +i)t M A(y~ 1 ) 

(11) =(k + l)\ s x/ ^x 1 ,...,x n -k-i)t M = (k + l)\S n , k {x;t). 

X,n\l(iu,)<k+1 

Using the fact that a a (u) = s„(u)as(u) for v — a — S p and Cauchy's formula for the sum of the Schur 
functions, second equation in ^w, we have 

v i<j 

In order for A(y~ x ) and ^ s M (y)t'^' to converge we need 1 < < li^ 1 ! for every i. For S nt k(x;t) to also 
converge, let \xj\ < 1 for all j. 

We also have that for any doubly infinite series f(y), [y°]f(y) — J c f(y)y~~ 1 dy, on a circle C in the C 
plane centered at and within the region of convergence of /. Rewriting A(y~ x ) in terms of y similar to ^ 
(where a sign of (— 1)( 2 ' appears) we obtain the formula for S n ^{x; t) through a complex integral. 

Proposition 2. We have that 



(k + l\ 



i<j<n—k—l J 

-1 / n (y*-vj) 2 TT 1 TT 1 TT — TT 1 < 

(2**)*+* J T 11 1 - t^y j 11 1 - Xl ty 3 11 1 - t Vi 11 Vi - 1 11 y iVj - 1 



_ -dy x ■ ■ ■ dyk+i, 

i<j<k+l l7ti7J i,j aJ aL aL i<j 

where T = C x x C 2 x • • • C p and C t = {z e C\\z\ = 1 + ej for e t < {t^l - 1. 



For the case of straight shapes we are looking for formulas for 

J2^(z)s x/l ,(x)tM 

A . 

with certain length restrictions for A and \i and certain values of x, z, t. Without the length restrictions we 
have by Cauchy's identity, (J3|) , that 

Z s k (z)s x /^x)s„(y) = s\(z)s x (x, y) = J] * ]J i * „ • 



Expanding the second product on the right-hand side using Cauchy's identity again we obtain 

E s *( z ) s vn( x ) s n(y) = I1 1 _\ - E s A z ) s Av) 

X.fi 



Comparing the coefficients for on both sides we get for any [i 

(12) E «#>/».(*) - II E «*•(*)■ 

A 3 V 

For the straight shape case we thus find 

D n , mik {x;z;t)= E a *(*) W^)*'** 1 = II i -z-g. 2 *f (*)*'"'. 

where the length restriction on A becomes redundant when evaluating the sum at x = (xi, . . . , x n -k-i) since 
then s\/n(x) = when l(X) > + n — fc — 1 < k + 1 + n — k — 1 = n. 

Proposition 3. We have that 

,{zt) 



D n , mi k{x,z;t) = H i _ 1 . ( E Sv{ ~ 

1,3 J \v\l{v)<k+l 



For the purpose of enumeration of SYTs we will use this formula as it is. Even though there are formulas, 
e.g. of Gessel and King, for the sum of Schur functions of restricted length in the form of determinants or 
infinite sums, they would not give the enumerative answer any more easily. 

5. A POLYTOPE VOLUME AS A LIMIT 

Plane partitions of specific shape (truncated or not) of size N can be viewed as integer points in a cone in 
R N . Let D be the diagram of a plane partition T, the coordinates of M' 15 ' are indexed by the boxes present 
in T. Then 

C D = {(••• ,Xi,j, ■ ■ ■ ) e M> : [i,j] e D, x hj < x iJ+1 if + 1] e D,x itj < x i+1J if [i + e £>} 
is the corresponding cone. 

Let -P(C) be the section of a cone C in IR> with the halfplane H = {x\J2[i j]eD x i-j — !}■ Consider 
the standard tableaux of shape T; these correspond to all linear orderings of the coordinates in Cp and 
thus also Pd = P{Cd)- Considering T as a bijection D — > [1, . . . , N], Pjj is thus subdivided into chambers 



{x : < Xx-i(i) < 2 ; t- 1 (2) < • • • a^T-ifJV)} H if of equal volumes, namely -^jVoZ(Ajv), where A^v = if fl k >0 



piV 

Ul Cl^UCll VU1LU11TO, 11CL111C1J v Ul\L-\JSj j 7 W11C1C L-l j\ li I IT 

is the TV-simplex. Hence the volume of Pd is 

(13) w ,_, w = glgM WM , 

The following lemma helps determine the volume and thus the number of SYTs of shape D. 

Lemma 1. Let P be a d-dimensional rational polytope in M> , such that its points satisfy a\ H h ad < 1 

for (ai, . . . , a,i) € P, and let 

= E E 

n (a u ...,a d )enPnZ d 

We have that the (i-dimensional volume of P is 

Vol d (P) = (lim(l- q ) d+1 F P ( q ))± 

Proof. Let / P (n) - #{(a u . . . ,a d ) G nP}. Then F P (g) - En/pW?" Moreover, Fo/ d (P) = lim„^ (X) ^ 
since subdividing an embedding of P in R d into (d)-hypercubes with side \ we get fp(n) cubes of total 
volume fp{n)/n d which approximate P as n — > oo. 



Moreover since lim n _ ) . 00 {n+1 y. {n+d} = 1 we also have Vol d (P) = lim n ^.oo ( „ + f ) F . ( "^ +d) ■ Let G(q) = 

En Th6n 

e (ff) = (i - g )G (ff ) = e ( -f^ - M /J f ^ n ) q - +d 

' (n + 1) • • • (n + a) (n) • • • (n + a — 1) 
and G(l) = J] 6„ = lim^oo b% H h 6 n = lim n _ ! . 00 ( n+ f)!.^ +d ) = Vol d (P). On the other hand 



Vol d {P) = G(l) = lim(l - q)G(q) = lim 

,-►1 (^)/ 



.. gW(g) .. f P (g)(l~g) d+1 
= • • • = hm , = lim — 

\ 1 — q 1 

by L'Hopital's rule. □ 

Notice that if P = P(Cd) for some shape D, then since (en, ... , ad) € mP when (<n, . . . , ad) € G25 and 
(Xi + • • • + ad < 771, we have that 

F P (q)= 9 ai+ - + °-(l + « + -) = T ^- E ? |B| = T^- E ^™ = ^-^(9). 

^-^ 1 — Q 1 — Q • <: — ' 1 — Q 

aGCn aeC D nZ N T:PP,sh(T)=D 



Using ( 13 1, the fact that Vol(A d ) — 4r and this Lemma we get the key fact to enumerating SYTs of truncated 
shapes using evaluations of symmetric functions. 

Proposition 4. The number of standard tableaux of shape D with N squares is equal to 

N\ lim(l -q) N F D (q). 

6. Standard tableaux of shape shifted staircase truncated by a box 

We are now going to use Propositions[2]and|4]to find the number of standard shifted tableaux of truncated 
shape 8 n \ 8\. Numerical results show that a product formula for the general case of truncation by dk does 
not exist. 

First we will evaluate the integral in Proposition [2] by iteration of the Residue theorem. For simplicity, 
let uq — t m = txi, so the integral becomes 

1 f (g/l - V2? 1 1 1 TT 1 , , 

/ .x 2 / i 7^ 7 7 T 11 1 ^l«2/2 

(2tti)^ y T l - ^2/12/2 2/2-12/1-1 yiy 2 - 1 i>0 ti 2 ~~ 

Integrating by yt we have poles at l,y% t~ 2 , y^ 1 and u^ 1 . Only 1 and y^ 1 are inside C\ and the respective 
residues are 

Res yi =! = - TT — 1 — ^ / ^ — \ 2 — r TT - — 1 dy 2 = 0, 

since now the poles for ?/ 2 are all outside G 2 . 
For the other residue we have that 

„ 1 f (V2 1 ~ V2? 1 1 1 TT 1 TT 1 , 

^ 27TZ i C2 1 - w - 1 - 1 y 2 11 1 - U ^ 2 11 1 - Ul y^ dV2 

= t 2 ^9 L / (! + y 2 ) 2 yr 3 n^— 111 — 1 — ^ 

If n > 3 the poles inside G 2 are exactly y 2 = Uj for all i and so we get a final answer 

-Erhi('+«0V j n^nr^ 



and 



s l<i<j<n— k — 1 J i>0 j^ti J ]>0 J 

where Ui — txi with Xo = 1. 

We can simplify the sum above as follows. Notice that for any p variables v — (vi, . . . , v p ) we have 

g TT(^) = rw^o = <»>) = S( ^ p+1)(w) = r " p+l(v) ' 

where h s (v) — 53i <i <—<« ^1^2 ' ' ' u i s i s the s-th homogenous symmetric function. 
Then we have that 

n—k—l g ^ n—k—1 s 

n-fe-1 ^s+p 
p>0 i=0 11V 1 3 ' 

(14) = ^h s ^ n+k+ i +p (u)h p (u) = c s _, i+fc+ i(u), 

p>0 

where Cj = X) n >o h n h n +i- We then have the new formulas 

(15) 5nil(:r;i ) = j|_I_ J] _l__2_ (ci(u) + Co(u))j 

1 \<i<j<n-k-l ' J 

where (1 + Wj) 2 u™~ 3 = u"~ 3 + 2u™ -2 + contributed to c_i(u) + 2c (m) + Ci(u) and by its definition 
c_i = c\. We are now ready to prove the following. 

Theorem 4. The number of shifted standard tableaux of shape 6 n \ Si is equal to 

C n C n —2 



+ 1 



fjn 



2G 



2n-3 



where g n — j-j — 2 (i+j) * s the number of shifted staircase tableaux of shape (n,n — 1, . . . , 1) and C m = 
--Vr f m ) is the m— th Catalan number. 

m+l \ m / 

Proof. We will use Proposition [4] and the formula (15). 

By the properties of <j) we have that for the shape D — S n \ 5f. , 

Fn(q)= £ £ g(»-*)l'*l + S^=5„ 1 *(«,g 2 > ... ig »- fc - 1 ;?»- fc ). 

T|sh(T)=13 P=<t>{T) 

Now let fc = 1. For the formula in Proposition [4] we have iV = ("J 1 ) — 1 and plugging x — (q,..., q n ~ 2 ), 
t = q' 1 ^ 1 in (15 1 we get 

Jm(l - q)"F D (q) = J] ^~ II x.^ x.in-D ^M + 

where u = (g n_1 , g™, . . . , g 2 ™ -3 ). 

We need to determine lim g _ J . 1 (l — g ) ™~ 3 c s (u). Let c s (x; y) = J2i hi(x)hi+ s (y) where x = (xi, . . . , x n ) and 
2/ = (2/1 > • • • j 2/m)- We have that 

c s (z; y) = X! E x h • ' ' ^ Vh ■ ■ ■ Vu+i 

l h<—<iv,ji<---<ji+B 
= Mvi> ...,Vp) E (-l) m+ - p - #p £ hi((xy) P ), 

P P:(l,p)-+(n,m) I 

10 



where the sum runs over all fully ordered collections of lattice points P in between (l,p) to (n,m) and 
(xy)p — {xi 1 yj 11 . . .) for . . . € P and the (— l)s indicate the underlying inclusion-exclusion process. 

We also have that, by 



Y\h{{xy)p) = ^ \ 

, Ll(i.i)ePv 



(»j)ep( 1 x iVj) 

The degree of 1 — q dividing the denominators after substituting (xi,yj) — (ui,Uj) — (q n ~ 2+i )q n ~ 2+ i) for 
the evaluation of c s (u) is equal to the number of points in P. #P is maximal when the lattice path is from 
(1, 1) to (n — 1, n — 1) and is saturated, so max(#P) = 2(n — 1) — 1 = 2n — 3. The other summands will 
contribute when multiplied by the larger power of (1 — q) and the limit is taken. For each maximal path 
we have {i + j \ G P} = {2, . . . , 2n — 2} and the number of these paths is ( "To ) > so 

■ x 2n-2 

2n — 4\ -p-r 1 - q 



(i-<» 2 -w ? 3 - 3 >= n rSb+c -«>■■■• 



i-2 ' 1 



where the remaining terms are divisible by 1 — q, hence contribute when the limit is taken 



Now we can proceed to compute Um„^i(l — q) N S n> i(q 1 , q 2 , . . . , ; q n 1 ). Putting all these together we have 



that 

\im(l~qfi 1 )- 1 S nA (q\...,q n - 2 : q n - 1 ) 



q-H- 



si n ^ n 1 1 _7 +J1 _ 1 ; ( n% (d- g ) 2 "- 3 ( C1 M +C o(.))) 



„i 11 1 _ 1 _ g,2(n-l) 

\<i<n-1 y l<i<j<n-2 * y 

tt 1 1 2 ( 2n ~ A \ 2 T\ 1 = jjg=j 1 /2n-4\ (2n-3)! 
o<ii-<«- 2 i + J ' 2 ( n_1 ) V"- 2 / tk 2n - A + 1 ( n 2 1 ) ! ( rl - 1 )U-2y (4n-6)!' 

where g„_2 = rf 2 r , ■■, is the number of shifted staircase tableaux of shape (n — 2, . . . , 1). After 

algebraic manipulations we arrive at the desired formula. □ 



7. Standard tableaux of shape rectangle truncated by staircase 

We will compute the number of standard tableaux of straight truncated shape D — n m \ 5k using propo- 
sitions [TJ [4] and [3} 

By Proposition [T] we have that 

F D (q)= J2 <? E ™ 

T:sh(T)=D 

^ q T,P[i,j]+i:Q[i,j]-W+{n-k)\n\ 
\,fj,\l(n)<k+l,l(\)<nP,sh(P),\/n,Q,sh{Q)=\ 

2 S x{l 1 q,q\...,q m - 1 ) S x/^q\---,q n - k - 1 )q [n - k)M , 
\,n\l{X)<n,l(n)<k+l 

which is -D n ,m,fc(#, z; t) for x = (q, q 2 , . . . , q n ^ k ^ 1 ), z = (1, q, . . . , q™ 1 " 1 ) and t = q n ~ k and from its simplified 
formula from Proposition [3] and Proposition [4] the number of standard tableaux of shape n™ \ 8u is 

(16) lim(l-< Z )"™-(^ 1 )F I? ((7) 

(n— k— l,m— 1 
n j-^fea -«r (fc+1) - ( * J1) ( E ^(5"- fc 1 ? n - fc+1 ) ---,? m - 1+ ™- fc )) 
i=l,j=0 " u\l{u)<k+l 

We are thus going to compute the last factor. 



Lemma 2. Let p > r and N = rp — (£) . Then for any s we have 



Km(l-q) N V s,(a 1+s p +»n g(P,P-i.-,P-r+D 



where 



E 1 (r, P ,s)= n , 9aW2 n 



,. 4 _ 2 (l + lsy/i 2 LL r ((l + 2s)(2p-l + 2 + 2s))lW 

for r even and Ei(r,p, s) = (^r^rr i)/2+s) ! ^ x ( r — ^'P' s ) wncn r i s °dd and g A is the number of shifted SYTs 
of shape A. 

Proof. Consider the Robinson-Schenstcd-Knuth (RSK) correspondence between SSYTs with no more than 
r rows filled with X\, . . . ,x p and symmetric p x p integer matrices A. The limit on the number of rows 
translates through Schensted's theorem to the fact that there are no m + 1 nonzero entries in A with 
coordinates . . . , (i r +i, j r +i), s.t. i\ < ■ ■ ■ < i r +i and ji > ■ ■ ■ > j r +i (i-e. a decreasing subsequence 

of length r + 1 in the generalized permutation corresponding to A). Let .4 be the set of such matrices. Let 
Ai C A be the set of — 1 matrices satisfying this condition, we will refer to them as allowed configurations. 

{1 if A\i j] ^ 
1 ,Ji r .We thus have that 
0, if A[i,j\ = 



y: -A(*i,...,*,)=Eii a; * ik 

A|f(A)<r J »>j 



oo 



= E n (£*n li (E(^) OM ) 

B€Bt:B[t,i] = l a«,i = l «>J':B[*J] = 1 o«j=l 

Err &i tt XiXj 

11 1x 11 1_ -r t 

BeBt:B[!,i] = l t>j:B[i,j] = l J 

Notice that B cannot have more than N nonzero entries on or above the main diagonal. No diagonal 
i+j — I (i.e. the antidiagonals) can have more than r nonzero entries on it because of the longest decreasing 
subsequence condition. Also if I < r or I > 2p — r + 1, the total number of points on such diagonal arc I — 1 
and 2p — I + 1 respectively. Since B is also symmetric the antidiagonals i + j = I will have r — 1 entries 
if I = r — l(mod2) and r is odd. Counting the nonzero entries on each antidiagonal on or above the main 
diagonal gives always exactly N in each case for the parity of r and p. If B has less than N nonzero entries, 
then 

\N TT TT q i +i+ 2s 



limu-^ n . n W+J+2S 



i:B[i,i] = l i>j:B[»,j]=l 
i:B[i,i] = l i>j:B[ij"] = l 

so such _Bs won't contribute to the final answer. 

Consider now only Bs with maximal possible number of nonzero entries (i.e. N), which forces them to 
have exactly r (or r — 1) nonzero entries on every diagonal i + j = I for r < I < 2p — r + 1 and all entries in 
i + j < r and i + j > 2p — r + 1. 

If r is even, then there are no entries on the main diagonal when r < I < 2p — r + 2 and so there are 
r/2 terms on each diagonal i+j = I. Thus every such B contributes the same factor when evaluated at 
x = (q 1 + s ,...,qP+ s ): 

q (l+2s)r/2 q (l+4s+2p-l+2)[l/2i 



E q (r,p,s):= [] n - „Wr/2 IT 



(l_ a l+2s)r/2 11 ((I _ Q l+2s\(l _ a 2p-l+2+2s))[l/2]- 
r<K2p-r+2 V H > 2<l<r KK 1 A H " 



If r is odd, then the entries on the main diagonal will all be present with the rest being as in the even case 
with r — 1, so the contribution is 

E q (r,p,s):= [I - — —^E q (r-l,p,s) 



r 4^<i<P-^- 



Let now M be the number of such maximal Bs in Aq. The final answer after taking the limit is MEi(r,p, s), 
where we have that Ei(r,p, s) — linig_>i(l — q) N E q (r,p, s) as defined in the statement of the lemma. 
In order to find M observe that the case of s = gives 

(17) Ml-?)" V s x (q\...,qP) = ME 1 (r,p,0), 

q— * — ' 

A|/(A)<r 

on one hand. On the other hand via the bijection cf> we have that 

X\l{X<m T 

where the sum on the right goes over all shifted plane partitions T of shape (p,p~ 1, . . . ,p— r+l). Multiplying 
by (1 — q) N and taking the limit on the right hand side gives us, by the inverse of Proposition |4j -^j times 
the number of standard shifted tableaux of that shape. This number is well known and is g(p,p_i,...,p_ r +i) = 
Y[ N 'h ) wnere the product runs over the hooklengths of all boxes on or above the main diagonal of (p r , r p ~ r ). 
Putting all this together gives 



9(p,p— r+l) 



MSi(r,p,0) 
Solving for M we obtain the final answer: 

9(p, P -i,...,p-r+i) E-i(r,p,s) 



Nl 



Ei(r,p,0y 



□ 



We can now put all of this together and state 
Theorem 5. The number of standard tableaux of truncated straight shape (n, n, . . . , n)\5k (assume n < m), 



(18) (mn-l 2 

where 

Ei(r,p,s) - 



)!x 



f(n-k-l) r ' 



9{ m ,m-x,...,m-k) E\{k + l,m,n — k — 1) 



(m(n-fc-l))! ((fc + l)m- ( fe + 1 ))! E^k + l,m,0) 

Il r <k2 P -t-+2 (l+2 S y/ 2 Il2<i<r ((Z+2s)(2p-Z+2+2s))Li/2J ' T even j 

f(r-l)/2+s)! 7T1 / i \ ij 
— '—Ei(r — l,p, s), r odd. 



(p-(r-l)/2+s)! J 



S + l 



Notice that the product Ei is just the product of the sum of s + l 
the coordinates i + j + 2s of the boxes (i, j) in the diagonal 
strip of width r above the main diagonal as in the picture 
below. When r is even the crossed diagonal squares 
for r < i < p — r + l are not included, and when r is odd 
they are included in the product as i + s. 

s + p 
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8. Truncation by almost squares 

We consider now rectangles truncated by a square without a corner, a case also considered in [AKRll] , 
We will use the methods developed so far to derive a formula for the volume generating function F£>(q) for 
these shapes and the number of standard truncated tableaux. 
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Specifically, let D = n rn \ (k k 1 , k — 1) be a rectangle truncated by square without a corner. For example, 
when n — 10, m = 8, k — 4, we have 



By symmetry we can assume that n < m. Consider the case when 2k < n + 1. For any plane partition 
T of shape D, let p = T[k, n — k + 1] be the value of the entry in that missing square corner. Because of 
the row and column inequalities, T is in bijection with T 1 of straight truncated shape n m \ <5fc_i, where the 
T'[i,j] — T[i,j] for G D and T'[i,j] = T[k,n — k + 1] = p for the values in the extra boxes. For example, 



T = 



G 


6 


■5 


4 








G 


5 


4 


3 








5 


3 


3 


2 


2 






4 


3 


2 


2 


2 


1 


1 


3 


2 


2 


1 


1 


1 


1 


2 


1 


1 


1 


1 


1 


1 



T' = 



6 


G 


■5 


4 


2 






6 


5 


4 


3 


2 


2 




5 


3 


3 


2 


2 


2 


2 


4 


3 


2 


2 


2 


1 


1 


3 


2 


2 


1 


1 


1 


1 


2 


1 


1 


1 


1 


1 


1 



Conversely, if T' is a plane partition of truncated shape n m \ S k -i whose n — k + 2nd diagonal /i = (T[l, n — 
+ 1], . . . , T[fc,n]) is of all equal entries, i.e. fj, = (p k ) for some p, this forces all entries T[i,j] = p for 

i > n — k + 1 and j < k. 

Then the generating function F nm \^ k k-i k _i^(q) is obtained, using equation |12| from 

(19) 



H n>m , k {z;x\t) = ^2^2s x (z)s x/{pk) (x)t p = Y[ -f— s {pk) (z)t p , 



by substituting a: = (q,q 2 ^ 



„n—k 



z = (l,g,. 



,q" 



as in the case of the rectangle truncated by a 



staircase n m \ For the value of t in this case, since X^I^j] = ~ Pi^^ 1 ) — *° accoun t f° r 

the extra squares in T', we have i» = g lMl(^-/ c +i)-p(( fc + 1 )-i) = ^{n-k+x^+^+iy _ Thcn 

^mVCfc*-!,*-!)^) = q, • ■ • , 9 m_1 ; 9, 9 2 , ■ ■ • , q n - k - q k ^- k H k Y) + ^ = 

m— /e 

(20) J] T^7+7 E S (P fc )(!' «' • • • . q m ^){q Hn - k) -^ +1 Y 

i=0;j'=l p 

The number of standard truncated tableaux of this shape will then be given by 



lim(l — q) 



nm—k +1 



F, 



n m \(fc'— 1 ,fc-l) 



(9)- 



We will now evaluate the sum over p and the relevant limit. 



Li<k<j 

,q 



Lemma 3. Let f q (v) = in 2 ) fj 

(21) X)*(p»)(1. 

p 

If t = q s , then 

j5i(i-ff)™*-*" +i x;«(p*)(i 



(vg* 



Y^ a i(q) v \ tnen 



m-l^p = 



n 



m— fc<2<m;0<j <m— fc 



(g* - ?J) 



gH 



i— l:m— k— 1 



L )(<f) ? 



n 



i 



(g) +s)!(fc(m-fc))! 

Proof. Consider the dcterminantal formula for the Schur functions, we have that 

s x (q°, . . . ,q m - 1 ) J] - qi) = det[(g 4 )^+"^] 4 = det^^) 4 ]^ = 



0<j<i<m 



(22) 



0<i<j<m+l 



where in our case A = (p k ,Q m k ). we have 

0<i<j<m+l 

0<i<j<k 0<i<k<j<m+l k<i<j<m+l 

(23) (gf)(5) J] (<,•»-*- J] (g™- - g"-i) JJ (gT"-^) 



Let 



0<i<j<k k<i<j<m+l 0<i<k<j<m+l 



A{q)= 1] (2 m -'-<r- i ) II {d m - i -q m - j ) 

0<i<j<k k<i<j<m+l 



which does not depend on p. The last equation, 23 is thus equal to A(q)f q (q p ), where f q (v) is the polynomial 
defined in the statement of the lemma. We have 

EW)^ = E "i(9)£?** p = E 



0<i<l p 0<i<l 



which together with A{q), equation (22) and some cancellations gives the first part of the lemma. For the 
limit we need to find 



limfl — g) aAa) : — . 

o<;<z y y 

This limit is equal to 

E fh{v)v°-Hv = 

' I + S /n 

o<i<; 1/0 

/ iASH'" 1 J] (vl™-* - l m - j )dv = vi^+^iv-l^-^dv 

•'° 0<i<fe<i<m+l 

For the last integral, it could easily be found recursively, e.g. if 

f 1 b 

J(o, 6) = / v a (i; - l) fc cfo, then /(a, 6) = I(a + 1,6- 1), 

Jo a + 1 

where I(a, 0) = -rr, so I(a,b) = (— l) b 7 : — : — ■ Noticing that lim TT — — — = TT — - — for 

v ' ; a+1 ' v ' ' v ; (a+1) ■■•(& + 6+1) 6 11 - g? 

the remaining mk — k 2 factors in equation (21 1 we obtain the desired limit. □ 
For the number of standard tableaux of shape D take the limit lim (1 — q^ nm - k +1 Fo(q) using the formula 

for Fu(q) from (20 1 and applying the second part of the Lemma with s — k(n — k) — (J 1 ) + 1. We can 
simplify the products by viewing them as products of hook-lengths and invoking the hook-length formula 
for the respective shape. 

Theorem 6. The generating function for truncated tableaux of shape D = n m \ (k k ^ 1 ,k — 1), where 
n > 2k — 1, is 

_^ _^ m— l;7i— k ^ 

fb(*)=5>(9) fcn _ fca _ m+1 n (7^7) n 13^+7- 

r 1 — <7 v y m-fc<i<m;0<7<m-fc vy y ; i=0;j=l y 

The number of standard truncated tableaux of that shape is 

{nm~k 2 + l)\f (mn - k) /(( m _ fc) fc) (fcn- fc 2 )!(fc(m- fc))! 
(m(n - k))\ ((m - k)k)\ {mk + fen - 2k 2 + 1)! ' 

After cancellations and regrouping of the factorials we obtain the binomial coefficient version of the 
formula in Theorem [3] from the Introduction. 

15 



Figure 1. The picture on the left above show the 3d Young diagram corresponding to 
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1 | 
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1 
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1 








1 











of shape (4,3,3, 1). If D = n" and T[l, 1] = n, as n —> oo, such plane 



partitions approximate stepped surfaces, as the picture on the right shows. 



9. 3-DIMENSIONAL YOUNG DIAGRAMS AND LOZENGE TILINGS 

In this section we will mention some connections with 3d diagrams/stepped surfaces/lozenge tilings, which 
are of interest as models in statistical mechanics. For an introduction to the subject see, e.g. |Ken09 . 

Plane partition T of shape D can be thought as a 3D diagram of unit cubes, see Figure |9j where the 
base in the xy plane is D and on top of each square G D there are T[i, j] unit cubes. Its volume is 
the number of cubes, i.e. ^2T[i,j], so our generating function Fjj(q) is the same as the volume generating 
function for plane partitions of base D: 

y Vol(T) 



F D {q) = 



T 



We usually draw 3D Young diagrams as their projection on the x + y + z = plane, where each side of 
the unit cube becomes a rhombus. The rhombi do not overlap, so the projection of T becomes a lozenge 
(pair of equilateral triangles) tiling of the triangular grid with certain boundary. 



Using the methods developed so far, in particular equation (12), we can easily derive volume generating 
functions for 3d Young diagrams with certain restrictions. Alternatively, these would correspond to lozenge 
tilings with free boundary. Namely, truncated plane partitions of shape n m \Sk, when drawn in 3 dimensions, 
are lozenge tilings of a hexagon with sides to, a,n,m — k,a,n — k, where a — > oo (i.e. height is unbounded). 
Here k of the lozenges on the right side will be "sticking out" , that is k of the triangles on the right boundary 
will be unmatched thus corresponding to a semi-free boundary condition, see Figure [9] 

Using the results we have obtained so far about Fo{q) for various shapes D we can state the following 
facts about the q vo1 distribution of the truncated 3d diagrams. Setting D = n n and fi = (b) in the formulae 
for Fjj(q) from the section on truncated rectangles we get the following 2 results. 

Proposition 5. The average value under the volume statistic of boxed plane partitions of base (n n ) and 
whose corner is at a fixed value b, i.e. T[l, n] — b, is given by 



J2T:sh(T)=n™;T ln =b 1 



ST:sh(T)=n" 
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Figure 2. The picture on the left represents the 3d diagram of a truncated plane partition 
with n = 8, m = 6, a = 5 and = 3. The middle picture shows the underlying triangular 
grid and the picture on the right shows the corresponding matching of the triangles in the 
lozenges, where each lozenge is the side of cube from the first picture. 



More generally, the volume generating function for plane partitions on (n n ), whose n — fc'th diagonal is 
fixed at a certain value n, i.e. (T[l, n — k + 1], T[2, n — k + 2], . . .) = /x, is derived as 



1 Xiyj 



for x = (1, ... , g™ 1 ), y — (q,q 2 , . . . , q n k ) and z = (<? n fc+1 , . . . , g n ). Notice that /x encodes the positions of 
the unmatched triangles ("sticking out" lozenges) in the lozenge tiling interpretation. Thus we have that 



E 



T:sh(T)=n n ;(T[l,n-fc+l],...,T[fc,n])=M 



TT r n m+n-i _ n fJ.j+n-j\ 
q T,T[i,j] = (Tt-fc+l)!^! iio<i<j<n+iW g ; 



lii+k-i _ „fij+k-j 



Uo<i<j<k(9 j ~ t) 



n 



0<i<n+l,0<j<n-fc+l 



To obtain the average distribution we can use the detcrminantal formula for the sum of Schur functions 
over partitions of restricted length [Ron King, unpublished notes] : 



^ s\(xi,...,x n ) = 

\,l(\)<rn 



1 



n.i<ji x i - ^jX 1 - x i x j) Uii 1 - x i) 



Aet^r (-l) m x(j > m)xr m+j -%=v 



Proposition 6. The distribution for truncated plane partitions with base n m and 
fixed rightmost diagonal = jjb is 

q\fi\(n-k) Y[ (1 — (J n_fe+ '~ 1 ) I! < \q^ i+m ~ i — q> 1 i +m ~i){\ — ( j 2 "- 2fe + i +J- 2 ) 



det[q^ n - k+i - 1 '> ( - m -^ - (-l) k+1 xti > k + l)g(n-fe+i-l)(m-fe+j-2)] 
We can also easily obtain the q vo1 distribution of ordinary plane partitions with base n x n and fixed 

n — fcth diagonal [i as in the picture 
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Proposition 7. The average wrt q Vo1 distribution of PP of shape n x n with a fixed n — fc'th diagonal 
(T[l,n-k + l],T[2,n-k + 2},...) = fj, is 



(»-k+i)H TT (g M -g" !2 X 

11 



0<i<j<n+l 



11 (n'J - „i\ 11 



(ql-q l ) 11 l-fl'+j-l" 

0<i<j<fc+l v ^ * y 0<i<n+l,n-fe<j<n+l ^ 
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